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Abstract. We consider a category of continuous Hilbert space representations 
and a category of smooth Frechet representations, of a real Jacobi group G. By 
Mackey's theory, they are respectively equivalent to certain categories of represen- 
tations of a real reductive group L. Within these categories, we show that the two 
functors of taking smooth vectors for G, and for L, are consistent with each other. 
By using Casselman-Wallach's theory of smooth representations of real reductive 
groups, we define matrix coefficients for distributional vectors of certain represen- 
tations of G. We also formulate Gelfand-Kazhdan criteria for Jacobi groups which 
could be used to prove the multiplicity one theorem for Fourier- Jacobi models. 



1. Introduction 

By a (complex) representation of a Lie group, we mean a continuous linear action 
of it on a complete locally convex space. In this paper, a locally convex space means 
a complex topological vector space which is Hausdorff and locally convex. When no 
confusion is possible, we do not distinguish a representation with its underlying space. 
Representations of reductive groups are studied intensively in the literature. These 
are the most interesting groups from the point of view of the Langlands program. 
But there is another family of groups which show up quite often in number theory, 
namely, Jacobi groups. 

We work in the setting of Nash groups. By a Nash group, we mean a group which 
is simultaneously a Nash manifold so that all group operations (the multiplication 
and the inversion) are Nash maps. The reader is referred to [Shi, Sh2] for details on 
Nash manifolds and Nash groups. Let G be a Nash group. A finite dimensional real 
representation of G is said to a Nash representation if the action map G x E ^ E 
is Nash. A Nash group is said to be almost linear if it admits a Nash representation 
with finite kernel. It is said to be unipotent if it admits a faithful Nash representation 
so that all group elements act as unipotent operators. Every unipotent Nash group 
is connected and simply connected. As in the case of linear algebraic groups, every 
almost linear Nash group has a unipotent radical, namely, a largest unipotent normal 
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Nash subgroup of it. (Recall that every Nash subgroup is automatically closed.) An 
almost linear Nash group is said to be reductive if its unipotent radical is trivial. 
Assume that G is almost linear, and denote by H its unipotent radical. The center 
of G is also an almost linear Nash group. Its unipotent radical Z is called the 
unipotent center of G. This equals to the intersection of H with the center of G. If 
G — 7- G is a finite cover of Lie groups. Then G is uniquely a Nash group so that the 
covering map is Nash. In this case, the unipotent radical (and the unipotent center) 
of G and G are canonically identified. 

As usual, we use the corresponding lower case German letter to denote the Lie 
algebra of a Lie group. We say that the almost linear Nash group G is a real Jacobi 
group if [f), f)] C 3 and the map 



is non- degenerate for some linear functional : 3 — ?■ M. By definition, every reductive 
Nash group is a real Jacobi group. It is easy to see that all real Jacobi groups are 
unimodular. 

Now assume that G is a real Jacobi group. Fix a unitary character ip on Z which 
is generic in the sense that the alternating form 



is non-degenerate, where dip denotes the differential of ijj. A representation of G is 
said to be a -j/^-representation if Z acts through the character ip. 

Among all representations, unitary ones are most important for many applications. 
But non-unitary representations also occur naturally even in the study of unitary 
ones. In general, we will consider ^-representations of G on Hilbert spaces so that H 
acts by unitary operators. Denote by T^modcvi the category of these representations. 
Morphisms in this category are G-intertwining continuous linear maps (which may 
or may not preserve the inner products). 

Smooth representations are also important for many purposes. In this paper, we 
are very much concerned with smooth Frechet -^/^-representations of G of moderate 
growth. (A representation is said to be Frechet if its underlying space is. See Section 
2.2 for the notion of smooth representations, and see Section 2.3 for the moderate 
growth condition.) Following F. du Cloux ([du4]), denote by J/'modc,^ the category 
of these representations. 

Given any representation V in 'HmodG t/j, it turns out (cf. Corollary 4.3) that its 
smooth vectors Vqo form a representation in jTmodc,,/, (see Section 2.2 for the notion 



[, ] : f)/3 X ^3 
is non-degenerate in the sense that the alternating form 

<^o[,]:f)/3xt)/3^M 




dip 



[,]:[)/3x[)/3^M 
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of smooth vectors). Therefore we get a functor (the smoothing functor) 

(2) ( ■ )oo : Hmodc,^ Jmodc,^. 

We are aimed to show that the smoothing functor (2) can be identified with the 
smoothing functor for a reductive group. 

To be more precise, use the alternating form (1), we form the real symplectic 

group Sp(f)/3) and its metaplectic double cover Sp(f)/3). The adjoint action induces 
a homomorphism 

L:=G/if-^Sp(t)/3). 

Define the fibre product 

L:=L Xsp(f,/3) Sp([)/3). 

This is a double cover of L. All the groups Sp([)/3), Sp(f)/3), L and L are canonically 
reductive Nash groups. 

A representation of L is said to be genuine if the nontrivial element in the kernel of 
the covering map L — )■ L acts as the scalar multiplication by —1. This terminology 
applies to other double covers of groups. Denote by "Hmodj; the category of 

genuine representations of L on Hilbert spaces, and by J'modj^^^^ the category of 

genuine smooth Frechet representations of L of moderate growth. As in the case of 
Jacobi groups, we still have the smoothing functor: 

(3) ( ■ )oo : Hmodi^^^ -> J mod^ g^^. 
Put 

G:=GxiL. 

This is a double cover of G so that G/H = L. In order to relate '0- representations 
of G to genuine representations of L, we fix a unitary oscillator representation w^, of 

G corresponding to ip: is a genuine unitary -j/^-representation of G which remains 
irreducible when restricted to H. Such a representation always exists (see Section 
4), and all others are unitarily isomorphic to twists of by unitary characters of 
L. Denote by the smoothing of u^. We construct in Section 4 four functors: 



(4) HomH(wJ, ■) : "Hmodcv. ^ T/mod^ g^^, 

(5) wj^h- : "Hmodj ^ "Hmodcv;, 

(6) Romn {uj^, ■) : Jmodc,^ -> Jmodi^^^^, 

(7) o;^®- : Jmodi^^^^ Jmodc ^p. 



The usual meanings of the topological tensor products "®h" and "®" will be ex- 
plained in Section 2.1. 

One purpose of this paper is to clarify the following 
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Theorem A. The functors (4) and (5) are inverse to each other, the functors (6) 
and (7) are inverse to each other, and the diagrams 

(■)c 



"Hmodc,^ 

Homff(w[;,,-) 



(■)c 



Homif(a;^,-) 



?/mod 



(■)c 



0,-4) 



(■)c 



commute. 



More precisely, Theorem A says that we have the following natural identifications 
of representations: 



a;J,§hHom^,(a;J,\/^^) 
Hom^^(a;J,wJgh^^') 

u^^Romniuj^, V) 
Hom//(w^, w^^E) 

HomH(c^5,V^'^)oo 



E, 



for all representations in Hmodc^, i?"^ in 'Hmodj^^^^, V in jTmodc^, and E 
in jT'modT . The first two assertions of Theorem A are in some sense well known 
(Mackey's theory, cf. [du4. Proposition 4.3.9]). They are rather direct consequences 
of the work of F. du Cloux ([du2, du3, du4]). The last assertion of Theorem A 
generalizes the expectation of R. Berndt in [Be, Page 185]. 



We say that a representation of a reductive Nash group is a Casselman-Wallach 
representation if it is Frechet, smooth, of moderate growth, admissible and Z-finite. 
Here Z is the center of the universal enveloping algebra of the complexified Lie 
algebra of the group. The reader may consult [Ca], [Wa, Chapter 11] and [BK] for 
more details about Casselman-Wallach representations. 

Denote by J^Tii the category of genuine Casselman-Wallach representations of 

L. This is a full subcategory of i/modj; gg^. Denote by J^Hg,^ the full subcategory of 
J^modc ,^ corresponding to J^'Hi^^^^, under the functors (6) and (7). Objects of this 
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category are called Casselman-Wallach ^/'-representations of G. These representations 
should be useful in the study of Jacobi forms (cf. [EZ]). It is important to note that 
the underlying spaces of all Casselman-Wallach ■^/'-representations are nuclear. 

As an application of Theorem A, we define matrix coefficients for distributional 
vectors in Casselman-Wallach ■^/'-representations, as what follows. Denote by C^{G) 
the space of tempered smooth functions on G, and by C^^{G) the locally convex 
space of tempered generalized functions on G. The later contains the former as a 
dense subspace (see Section 2.3 for precise definitions of these spaces). 

Let U he a representation in J^Tic^^ and let ^ be a representation in J^'Hg.'^i which 
are contragredient to each other, namely, a non-degenerate G-invariant continuous 
bilinear form 

(,):[/ X 1/ ^ C 

is given. Note that every representation in Fl-Lcip has a unique contragredient rep- 
resentation in J^I-Lq ,^ (Proposition 5.3). Denote by f/~°° the strong dual of V . It is 
a smooth representation of G containing [/ as a dense subspace. Similarly, denote 
by the strong dual of U . 

For any n G f/, f G the (usual) matrix coefficient Cu(^v is defined by 

(8) Cu(^^{g) := {gu,v), geG. 

It is a function in C^{G). The following result is proved in [SZl, Theorem 2.1] for 
real reductive groups. 

Corollary B. With the notation as above, the matrix coefficient map 
(g) UxV ^ C«(G), 

(u, v) I— >■ Cui2)v 

extends to a continuous bilinear map 

jj-co ^ y-oo _^ Cr^^G), 

and the induced G x G intertwining linear map 

(10) c : ^ cr^iG) 

is a topological homomorphism with closed image. 

Recall that a linear map cf) : E ^ F oi locally convex spaces is called a topological 
homomorphism if the induced linear isomorphism i?/Ker((/)) — Im(</)) is a homeo- 
morphism, where i?/Ker(0) is equipped with the quotient topology of -E, and the 
image lm(0) is equipped with the subspace topology of F . The action of G x G on 
C~^{G) is given by 

((^1,^2)-/) (a;) := f{g2^xgi). 
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In particular, Corollary B defines characters of Casselman-Wallach ^-representations 
(as tempered generalized functions on G). It also implies that irreducible Casselman- 
Wallach -i/^-representations are determined by their characters ([SZl, Remark 2.2]). 

The following form of Gelfand-Kazhdan criteria is a rather direct consequence of 
Corollary B. See [SZl, Theorem 2.3] for a proof. 

Corollary C. Let Si and S2 be two closed subgroups of the real Jacobi group G, with 
continuous (non-necessarily unitary) characters 

Xi:^. ->C^ 2 = 1,2. 

Assume that there is a Nash anti- automorphism a of G such that for every f G 
Cr^{G) which is an eigenvector ofU^Qc)^, the conditions 

f{zx) = ip{z)f{x), 

f{sx) = xi{s)f{x), s e Si, and 

f{xs) = X2{s)-^f{x), s e S2 

imply that 

fixn = fix). 

Then for every pair of irreducible representations U in TTici) and V in J^'Hq^^ which 
are contragredient to each other, one has that 

dimHom5j(f/, Xi) dimHom5'2(V, X2) < 1- 

Here and henceforth, a subscript "C" indicates the complexification of a real Lie 
algebra, the universal enveloping algebra U(0c) is identified with the algebra of left 
invariant differential operators on G, and U(0c)*^ is identified with the algebra of 
bi-invariant differential operators on G. 

Another purpose of this paper is to have the following criterion for a strong Gelfand 
pair, which is used in [SZ2] to prove the multiplicity one theorem for Fourier- Jacobi 
models. As in the proof of [SZl, Corollary 2.5], the criterion is implied by Corollary 
C. We shall not go to the details. 

Corollary D. Let G' be a Nash subgroup of G which is also a real Jacobi grouop. Fix 
a generic unitary character ip' of the unipotent center Z' of G' . Assume that there 
exists a Nash anti- automorphism a of G preserving G' with the following property: 
every tempered generalized function on G which is invariant under the adjoint action 
of G' is automatically a -invariant. Then for every irreducible Casselman-Wallach 
ip -representation U ofG, and every irreducible Casselman-Wallach ip' -representation 
U' of G' , the space of G' -invariant continuous bilinear functionals on U x U' is at 
most one dimensional. 
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2. Preliminaries 

In this section, we fix some notations and terminologies and recall some general 
results which will be used later in this paper. 

2.1. Topological tensor products. Let E and F be two locally convex spaces. 
There are at least four useful locally convex topologies one can put on the algebraic 
tensor product E ® F, namely, the inductive tensor product E ®i F, the projective 
tensor product E F, the epsilon tensor product E ®^ F and the Hilbert tensor 
product E (8)h F. As usual, their completions are denoted by E®\F, E(S)nF, E®^F 
and E^^F, respectively. The inductive tensor product plays no role in this paper. 
The projective tensor product and the epsilon tensor product are more commonly 
used and the reader is referred to [Th] for a concise treatment. A fundamental 
theorem of Grothedieck says that they coincide when either or F is nuclear. If 
this is the case, we simply write 

E^F := E®^F = E^,F. 

We are more concerned with the Hilbert tensor product. Its topology is defined 
by the family 

of non-negative Hermitian forms on E ^ F, where ( , )^ and {, )u runs through all 
non-negative continuous Hermitian forms on E and F, respectively. 

Recall that a locally convex space is said to be Hilbertizable if its topology is 
defined by a family of non- negative Hermitian forms on it. The following fact is well 
known. 

Lemma 2.1. If E is nuclear and F is Hilbertizable, then 

E§)tF = E®F 

as locally convex spaces. 

Proof. We indicate the idea of the proof for the convenience of the reader. In general, 
the Hilbert topology is coarser than the projective topology. If both E and F are 
Hilbertizable, then the epsilon topology is coarser than the Hilbert topology. The 
lemma then follows by Grothedieck's theorem and the well know fact that every 
nuclear locally convex space is Hilbertizable (cf. [Th, Corallary 3.19]). □ 
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2.2. Smoothing representations. In this subsection, G is an arbitrary Lie group. 
Let y be a representation of G (recaU from the Introduction that every representation 
space is assumed to be complete). It is said to be smooth if the action map G^V ^ V 
is smooth as a map of infinite dimensional manifolds. The notion of smooth maps in 
infinite dimensional setting may be found in [GN], for example. In general, denote 
by C(G; V) the space of continuous functions on G with values in V . It is a complete 
locally convex space under the usual topology of uniform convergence on compact 
sets, and is a representation of G under right translations: 

{g.f){x):=f{xg), g, x E G, f E C{G;V). 

Similarly, smooth V^- valued functions form a complete locally convex space C°°{G\ V) 
under the usual smooth topology, and is a smooth representation of G under right 
translations. 
Write 

iv{.g) ■■= g-v, V eV, g EG. 

The smoothing Voo of V is defined to be the representation which fits to a cartesian 
diagram 

Kc > C^iG;V) 



V C{G-V) 

in the category of representations of G. This is a smooth representation of G. As a 
vector space, it consists of all f G such that 7„ E C^{G; V). The topology of Voo 
coincides with the subspace topology of C^{G; V). The universal enveloping algebra 
U(0c) Ei-cts on Voo as continuous linear operators by 

X.v := the value of ^(7,;) at the identity element 1 E G. 

The topology of Voo is determined by the family 

{| ■ U,A}xeU(gc),AeA 

of seminorms, where A is the set of all continuous seminorms on V, and 



\v\x,x ■■= \X.v\ 



A- 



The smoothing is clearly a functor from the category of representations of G to 
the category of smooth representations of G. When V itself is smooth, we have that 
V = Voo as representations of G. In general, Voo is a dense subspace of V. 

Lemma 2.2. Let Vq be a G-stable subspace of Voo- If is dense in V , then it is also 
dense in Voo- 
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Proof. Fix a left invariant Haar measure dg on G. Let / G C^{G) (a smooth function 
with compact support). For all v E V, write 



f-v ■■= / f\g)g.vdg. 
Jg 

It is well know and also easy to see that f.v G Vqo and the linear map 

V ^Voo, V f.v 

is continuous. Then the denseness of Vq in V implies the denseness of f.Vo in f.V, 
under the topology of Voo. Consequently, under the topology of Voo, 

(11) f-^o is dense in ^ f.V. 

Then the lemma follows by noting that the closure of Vq (within ^00) contains the 
first space of (11), and that the second space of (11) (the Carding subspace) is dense 
in Voo- 

□ 

For every closed subgroup S of G, write V\s '■= V, viewed as a representation of 

S. 

Lemma 2.3. Let S and S' be two closed subgroups of G. If the actions o/U(sc) and 
U(5c) produce the same subalgebra o/End(Voo), then 

{V\s)oo = {V\s>)oo 

as locally convex spaces. 

Proof. Apply Lemma 2.2 to the representation V^l^, we see that Voo is dense in 
iy\s)oo- Therefore {V\s)oo is the completion of Voo under the seminorms 

{| ■ U,A}xeu(sc), AeA- 
Similarly, (l^|5')oo is the completion of Voo under the seminorms 

{| ■ U,A}xgU(s[.), AgA- 

The assumption of the lemma implies that these two families of seminorms are the 
same. Therefore the lemma follows. 

□ 

We say that a representation is Hibertizable if its underlying space is. The smooth- 
ing of a Hibertizable representation is also Hibertizable. Let V be another repre- 
sentation of another Lie group G' . Then V®t^V' is a representation of G x G'. If 
both V and V are Hibertizable, then V®\y is also a (Hilbertizable) representation 
of G X G'. 
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Lemma 2.4. // both V and V are smooth, then so is Vi^t^V . If both V and V are 
smooth and Hibertizable, then so is V®}iV' . 

Proof. We prove the firs assertion. The second one is proved similarly. The algebraic 
tensor product V ® V is clearly contained in {y®T,V')oo- It is dense in V®.„V' 
and is therefore also dense in {y®T,V')cxi, by Lemma 2.2. Note that U(gc x acts 
continuously on V ®t,V' . This implies that the topology of {y®TrV')oo and of V®t,V' 
have the same restriction to ® V . Consequently, {y®.„V')oo and V®t,V' are both 
completions oiV ®V' with respect to a common locally convex topology. Therefore 
they are the same and the first assertion of the lemma follows. □ 

Lemma 2.5. // both V and V' are Hibertizable, then 

(12) (^®hOoo = V^®^Vi,. 

Proof. The proof is similar to that of Lemma 2.4 and we will not go to the details. 
The key point is that both sides of (12) contain V^o ® K>o a dense subspace. □ 

It is not clear to the author whether the analog of (12) holds for projective tensor 
products. 

2.3. Representations of moderate growth. We will use the notation of [SZl] for 
function spaces, as what follows. Let G be an almost linear Nash group. It is not 
assumed to be a real Jacobi group in this section. We say that a (complex valued) 
function f on G has moderate growth if its absolute value is bounded by a positive 
Nash function (j) on G: 

\f{x)\ < (f){x) for all X e G. 
A smooth function / G C^{G) is said to be tempered if Xf has moderate growth 
for all X e U(0c)- Denote by C^{G) the space of all tempered smooth functions on 
G. 

A smooth function / G C^{G) is called Schwartz if 

\f\x,4> := sup^^G^Pi^) \i^f)ix)\ < oo 

for all X G U(gc), and all positive functions on G of moderate growth. Denote by 
C''{G) the space of Schwartz functions on G. It is a nuclear Frechet space under the 
seminorms {| -Ix,!/)}- We define the nuclear Frechet space D''(G) of Schwartz densities 
on G similarly. Fix a Haar measure dg on G, then the map 

C'{G) D^(G), 
f ^ fdg 

is a topological linear isomorphism. We define a tempered generalized function on 
G to be a continuous linear functional on D''(G). Denote by C~^{G) the space of all 
tempered generalized functions on G, equipped with the strong dual topology. This 
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topology coincides with the topology of uniform convergence on compact subsets of 
D'^(G'), due to the fact that every bounded subset of a nuclear locally convex space is 
relatively compact. Note that C^{G) is canonically identified with a dense subspace 

of cr^{G): 

A representation of G is said to be of moderate growth if for every continuous 
seminorm | ■ |^ on there is a positive function on G of moderate growth and a 
continuous seminorm | ■ |j, on such that 

\9'v\fi < (p{g) \v\u, for all G G, G V. 

It is easy to check that the smoothing of a representation of moderate growth is 
still of moderate growth. For every representation ^ of G of moderate growth, the 
bilinear map 



D^(G)xV ^ 
>^ = f{9)dg,v H-> X.v:= J^f{g)g.vdg 



(13) " " ' 



is well defined and continuous. The space D^(G) is an associative algebra under the 
convolution operator *. Under the map (13), both V and Vqo are D^(G)-modules. 

2.4. Matrix coefRcients for distributional vectors. Let G be an almost linear 
Nash group as in the last subsection. Let and be two Hilbert space repre- 
sentations of G of moderate growth which are strongly dual to each other in the 
following sense: there is given a G-invariant continuous bilinear map 

( , ) : t/^ X ^ C 

which induces a topological isomorphism from to the strong dual of (and 
hence a topological isomorphism from V"^ to the strong dual of f/^). Denote by 
the strong dual of the smoothing of V"^. This is a locally convex space carrying 
a linear action of G. There are canonical G-intertwining continuous injections 

t/^ ^ t/^ ^ u-^. 

It is not clear to the author whether is dense in ?7~°° in general. Similarly, denote 
by the strong dual of the smoothing f/^. 

Define a bilinear map 

D^(G)xf/— ^ f/^^ = Homc(F^C), 
^ ^ \ = J{g)dg,<\> ^ X4:={v^<P{J^f{g)g-\vdg)). 

The following lemma is elementary and may be proved by the argument of [SZl, 
Section 3]. 
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Lemma 2.6. The map (14) is well defined and its image is contained in U^. The 
resulting bilinear map 

W{G) X U~°^ Uoo, (A, 0) ^ A.0 

is separately continuous and extends the continuous bilinear map (13) for the repre- 
sentation U^. 

Finally, we define the matrix coefficient map 

c : U-^ X V-°^ Cr^{G) = Homc(D^(G), C), 
(j), (p' ^ c<^®0/ := (A t-^ 0'(A-0)) • 

This extends the ordinary matrix coefficient map (see (8)), and is separately contin- 
uous (see the proof of [SZl, Lemma 3.6]). 

3. Representations of Heisenberg groups 

Let w be a finite dimensional real vector space with a non-degenerate skew- 
symmetric bilinear map 

( , )„ : w X w — z, where dimiK(z) = 1. 

Let H(w) = w X z be the associated Heisenberg group, with group multiplication 

{u,t){u',t') := {u + u',t + t' + (m,m')w)- 

Let ip^ be a non-trivial unitary character on z. By Stone- Von Neumann Theorem, up 
to isomorphism, there is a unique irreducible unitary -j/^z-representation of H(w). 
Write u^^ for its smoothing. It is well known that this is a nuclear Frechet space. 

3.1. Smooth representations. Recall that i7modH(w),i/iz is the category of smooth 
Frechet V'z-representations of H(w) of moderate growth. The following fact is funda- 
mental to this paper. 

Proposition 3.1. For every representation V in j7'modH(w),?/>z; HomH(w) (i^V'z ; ^) ^■^ 
a Frechet space under the topology of uniform convergence on bounded sets. For 
every Frechet space E, uj^^®E is a representation in j7'modH(„),^^ . Further more, 
one has that 

u^^ ® HomH(w) (w^z ,V) =V, 

and 

HomH(w)('^^z7^5/'z®-^) = E. 

Consequently, the functors 

HomH(vy)(w^z5 ■) ^^^^^ w^z®" 

are mutually inverse equivalences of categories between the category j7'modH(„),^^ 
and the category of Frechet space. 
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In view of the following lemma, Proposition 3.1 is a combination of [dul, Theorem 
3.3] and [du3, Theorem 3.4]. 

Lemma 3.2. Denote by I^^ the annihilator of u^^ in D'^(H(w)). Then I^^ annihi- 
lates every representation in j7'modH(w),V'z ■ Consequently, every representation in 
jrmodH(w),?/.z is a 'D'^(R{w))/Ltp^- module. 

Proof. Fix a Haar measure dw on w and a Haar measure dz on z. Then dh := dw®dz 
is a Haar measure on H(w). Define a continuous surjective map 

D^(H(w)) ^ C^(w), 
^ ' \ = fdh^{w\-^ J^f{zw)ilj{z)dz). 

Denote by J^^ the kernel of this map. It is checked to be a closed ideal of D'^(H(ty)). 
We view C"'(w) as an associative algebra so that the map (15) is an algebra homo- 
morphism. It is easy to see that all representations in i/modH(w),V'z annihilated 
by /^^. Therefore they are all C""(w)-modules. 

On the other hand, a classical result of I. Segal says that the action of C"'(w) on 
u!^^ is faithful (c.f. [Ho, page 826]). Therefore /^^ = I^^. This proves the lemma. 

□ 

3.2. Unitary representations. As usual, for every complex vector space E, write 
E for its complex conjugation. This equals to as a real vector space, and its 
complex scalar multiplication is obtained by composing the complex conjugation 
with the scalar multiplication of E. 

Denote by ( , ),/,^ the inner product on the Hilbert space u^^. It restricts to a 
nonzero H(w)-invariant continuous bilinear form 

( , : X ^^z ^ C. 
It is well known that such a form is unique up to scalar (see [du4. Proposition 4.12] 
for example). This implies the following 

Lemma 3.3. For all Frechet spaces E and F , the map 

B{E,F) Bh(w)(w^z®^,^^z®^), 
b ^ ( , )^z ® & 

is a linear isomorphism, where 'B " stands for the space of continuous bilinear forms, 
and "Bh(w) " stands for the space of -invariant continuous bilinear forms. 

Denote by 'HmodH(w),i/)z the category of unitary '^^-representations of H(w). For 
every Hilbert space E with inner product (,)£;, 0J^^®TaE is a unitary representation 
in 'HmodH(„),,/,^. (The inner product is ( , ® ( , )e-) Conversely, we have 

Lemma 3.4. Every representation V in 'HmodH(w),V'z ^■^ unitarily isomorphic to 
ujI ®\yE for some Hilbert space E. 
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Proof. This is well known. We provide a proof for completeness. By Proposition 
3.1, the smoothing Voq has the form u^^^Eq, where Eq is a certain Frechet space. 
By Lemma 3.3, the restriction of the inner product on V to uj^J^iEq has the form 
( , ® fc, where h is an inner product on Eq. Denote by E the completion of Eq 
with respect to h. This is a Hilbert space and we have that V = cj^^®h-E'- 

□ 

For every unitary representation V in 'HmodH(w),V'z5 with inner product (, )v, 
define an inner product on HomH(w)(i^{i5 V) by 

(16) (0,0') := 0'* o G HomH(w)«,a;;j = C, 

where 0'* G B.omii(w){V, ^^,J is determined by the formula 

{(l)'{u),v)v = (m, (p'*iv))^^, ueuj'^^,vEV. 

The following analog of Proposition 3.1 is an easy consequence of Lemma 3.4. 

Proposition 3.5. For every representation V in 'HmodH{w),^/)2; HomH(w) (i^l/,^ , ^) is 
a Hilbert space under the inner product (16). For every Hilbert space E, u^^®\yE is 
a representation in 'HmodH(w).?/)z ■ Further more, one has inner product preserving 
identifications 

<®hHomH(w)(<,V^) = V^, 

and 

HomH(w)(w{„,wJ,®h^) = E. 

Consequently, the functors 

HomH(vi/)(wJ^, ■) and wj^ih" 

are mutually inverse equivalences of categories between the category 'HmodH(w),i/'z 
and the category of Hilbert spaces. 

4. Representations of Jacob: groups 

Let G be a real Jacobi group, and we return to the notation of the Introduction. 
This section is devoted to a proof of Theorem A. Without lose of generality, we 
assume in this section that ip has a discrete kernel. Then Z is either trivial or one 
dimensional. If it is trivial, then H is trivial and Theorem A is also trivial. So further 
assume in this section that Z is one dimensional. Then if is a Heisenberg group as 
in last section. 
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4.1. Splitting Jacobi groups. Fix a subspace w of the Lie algebra f) which is 
complementary to 3. Such a space is unique up to conjugations by H. It determines 
a Nash splitting : L ^ G oi the quotient map G — >■ L so that iiiL) stabilizes w 
under the adjoint action (cf. [Mo, Theorem 7.1]). Identify L with iiiL), then we 
have 

G = Li< H. 

As in the Introduction, w = f)/^ is a symplectic space under the Lie bracket 

[, ] : w X w ^ 3. 

We let the symplectic group Sp(w) acts on H as group automorphisms so that it 
point-wise fixes Z and induces the natural action on w. Let Sp(w) acts on H through 
the covering map Sp(w) — )■ Sp(w). Then the following semidirect product is a real 
Jacobi group: 

J(w) := Sp(w) K H. 

The adjoint action induces a homomorphism L — )■ Sp(w). Recall from the Introduc- 
tion that 

L := L Xsp(w) Sp(w) 
is a double cover of L. The double cover 

G ■.= € XlL = Lk H 

of G is obviously mapped to both J(w) and L. In this way, we view it as a subgroup 
of the product J(w) x L. 

4.2. The oscillator representation. To distinguish representations of different 
groups, we write 7t\s to emphasize that vr is viewed as a representation of a group 
5". Denote by the unitary oscillator representation of J(w) corresponding to 
ijj. Up to isomorphism, this is the only genuine unitary ^-representation of it which 
remains irreducible when restricted to H. Without lose of generality, assume that 
the representation of G in the introduction coincides with the pull back of to'^lj(^) 

to G. 

The following lemma is well know (see [Ad, Section 2], for example). 

Lemma 4.1. The universal enveloping algebras U(5p(w)c x ()c) and U(f)c) produce 
the same subalgebra c)/End((a;^|j(-^-))oo)- 

Lemma 2.3 then implies that 

:= (W^lg)oo = (^V'lj(w))«^ = (w^|/f)oo 

as Frechet spaces. 
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4.3. Equivalences of categories. Recall from the Introduction the categories 
nmodci,, Jmodc,^, Hmodj g^^, and Jmodi ^^^. 

Given any representation V in Hmodc,,/,, recall from the last section that Homfj^u^, V) 
is a Hilbert space. Let G act on it by 

:= g{(l){r^x)), geG,(Pe HomH(u;J, V^), x G wj, 

where g is the image of g under the quotient map G ^ G. This action is checked to 
be continuous (use Proposition 3.5) and descends to a genuine representation of L. 
Therefore we get a functor 

Hom//(w{,, ■) : nmodci, "Hmodj g^^. 

On the other hand, given any representation E in Hmodr , the tensor product 

^^hE is a Hilbert space representation of J(w) x L. Its restriction to G descends 



to a representation of G. In this way, we get a functor 

wj^h ■ : "Hmod^ -> "Hmodc,^. 
Similarly, we have functors 

(17) Romniuj^, ■) : J^modc,^ Jmodj 
and 

(18) u^,® ■ : Jmodj g^j^ J^modc,^. 
Proposition 4.2. The functors 

Hom//(a;^, ■) and a;^®h ■ 

are mutually inverse equivalences of categories between "HmodG^^ and Timodj^ ^g^. 
Similarly, the functors 

Homnico^, ■) anc? w^® ■ 
are mutually inverse equivalences of categories between jTmodc^^ and jTmodj; 

Proof. In view of Proposition 3.5, to prove the first assertion, it suffices to show that 
the identification 

HomH(M/)(a;5,,cuJ®h^) = E 
respects the L-actions, and the identification 

respects the G-actions. This is routine to check and we will not go to the details. 
The second assertion is proved similarly. □ 

Corollary 4.3. Every representation in "Hmodc,^ is of moderate growth. 
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Proof. Since every representation in 'HmodG'^ has the form u^(S)hE {E is a represen- 
tation in Hmodj; g^j^), this follows from the well know fact that every Banach space 
representation of a reductive Nash group is of moderate growth (cf. [Wa, Lemma 
11.5.1]). □ 

4.4. Smoothing. Recall that G is a Lie subgroup of J(w) x L, and therefore 0c is 
a Lie subalgebra of (sp(w)c x f)c) x fc- 

Let -E be a representation in "Hmodr . Write 

viewed as a representation of J(w) x L. The descent to G of its restriction to G is 
denoted by V . Lemma 2.5 and Lemma 2.1 implies that 

= (^^lj(„))oo®h-Eoo = tu^|j(^)®h-Eoo = a;vlj(w)®-^oo- 

Lemma 4.4. The universal enveloping algebras U((sp(w)c x f)c) x fc) o,nd U(0c) 
produce the same subalgebra o/End(VJ^). 

Proof. Write 

p : U((sp(w)c X f)c) X Ic) ^ End(y^) 
for the Lie algebra action. Lemma 4.1 implies that 

p(sp(w)c) C p(U(f}c)) C p(U(0c))- 
The lemma then follows by noting that 

(sp(w)c X [)c) X t = 5p(w)c + 0c- 



□ 



Finally, by Lemma 2.3, we conclude that 
Together with Proposition 4.2, this finishes the proof of Theorem A. 



5. Casselman- Wallace ^'-representations and generalized matrix 

coefficients 

We continue to use the notation of the Introduction and assume that G is a real 
Jacobi group. 
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5.1. Casselman-Wallach ^/'-representations. Recall from the Introduction that 
^'Hg,ip is the full subcategory of Juiodc^ip corresponding (under the functors (6) 
and (7)) to the subcategory J^'H^^^^ of jTmodj^g^^. Objects in J^Tic^p are called 
Casselman-Wallach ^/'-representations of G. 

The following lemma is well known and is an easy consequenc of Casselman- 
Wallach's Theorem ([Wa, Corollary 11.6.8]). 

Lemma 5.1. Every injective homomorphism cf) : E ^ F in the category J^'H^ is 
a topological embedding with closed image. 

We use the above lemma to show the following 

Lemma 5.2. Let U be a smooth Frechet representation ofG of moderate growth, and 
let V be a Casselman-Wallach ip -representation of G. Then every G -intertwining 
continuous linear map from U to V is a topological homomorphism with closed 
image. 

Proof. The map descends to an injective homomorphism 

0' : U/KeT{(f)) V 

in the category jTmodG,^. By Proposition 4.2, this corresponds to an injective 
homomorphism 

(p" -.E ^F 

in the category J'modi^^^^. Since F is in J^'Hi^^^, by applying Harish-Chandra's 
functor to 0", we see that E is also in ■F'Hi^^^^. Then Lemma 5.1 says that 0" is 

a topological embedding with closed image. This implies that so it 0', since oo^® ■ 
is a topologically exact functor on the category of Frechet spaces (cf. [Ta, Theorem 
5.24]). □ 

Now let U = uo^®E be a representation in Tl-Lcipi and let V = uj^^F be a 
representation in J^Tic,^- Here both E and F are representations in J^'H^^^^. We 
say that they are contragredient to each other if there is given a non-degenerate 
G-invariant continuous bilinear form 

{,):UxV ^C. 

Proposition 5.3. Every representation in J^l-Lcip has a unique contragredient rep- 
resentation in J^T-Lg,^- 
Proof. Lemma 3.3 implies that 

Bg{U,V) = Bi{E,F). 

Therefore, in view of Proposition 4.2, the Proposition follows from the corresponding 
result in the category J^l-ii But the later is a consequence of Casselman-Wallach's 
Theorem. □ 
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5.2. Generalized matrix coefficients. Let U, V, E, F be as before. Assume that 
U and V, and hence E and F, are contragredient to each other. It is well known 
that there is a representation E^ in "Hmodj so that its smoothing coincides E 

(cf. [BK, Section 3.1]). Denote by F^ the space of continuous linear functionals on 
E^. As usual, it is a representation in "Hmod^gg^ which is contragredient to E^. 
Put 

[/'^:=a;J§h^^ and := wJ®hi^^ 

View them as representations in "Hmodc ,^ and 7/modg^, respectively. Theorem A 
implies that 

Ul = U and V^ = V. 

Denote by U~°° the strong dual of V, and by V"°° the strong dual of U. They are 
both representations of G (due to the fact that both U and V are Nuclear Frechet, 
and hence reflexive spaces). Furthermore, f/~°° contains [/ as a dense subspace, and 
V~°° contains ^ as a dense subspace. 

Note that and are strongly dual to each other. By the argument of Section 
2.4, we get a separately continuous bilinear map 

(19) u-°° X V^°^ cr^{G) 

which extends the usual matrix coefficient map. Note that all the three spaces in (19) 
are strong duals of reflexive Frechet spaces. Therefore [Tr, Theorem 41.1] implies 
that the map (19) is automatically continuous. Then (19) induces a continuous linear 
map 

(20) c : f/-~§V^-°° ^ Cr^{G). 

By [Tr, Proposition 50.7], U~°°^V~°° equals to the strong dual of V(S)U. Therefore 

(20) is the transpose of a G x G- intertwining continuous linear map 

(21) D'{G)^V^U. 

Apply Lemma 5.2 to the group G x G, we see that the map (21) is a topological 
homomorphism with closed image. In view of the following lemma, we conclude that 
(20) is also a topological homomorphism with closed image. This proves Corollary 
B. 

Lemma 5.4. (See [Bo, Section IV. 2, Theorem 1] and [SZl, Lemma 3.8]j Let : 
El E2 be a continuous linear map of nuclear Frechet spaces. Denote by E[ and 
E2 the strong duals of Ei and E2, respectively. Then (p is a topological homomorphism 
if and only if its transpose 0* : i?2 — >■ E[ is. When this is the case, both and 0* 
have closed images. 
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